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ABSTRACT 

This paper compares the visual and auditory 
perceptual systems as bases for communication systems, reviews 
advances in computer technology, and considers implications for 
literacy among persons with hearing impairments. It is pointed out 
that the primary language skills of hearing individuals are developed 
through speech and language (auditory perception) and then 
transferred to the written form (visual perception), and that visual 
communication systems inherently contain some ambiguities (amon^ them 
the irregularities of English spelling). Studies suggesting that the 
printed word is the most stable of the visual sjonbols used by 
hearing-impaired children are noted. Technology is seen as an 
important potential ally in the language development of children with 
hearing impairments, through providing new and different experiences 
that associate language symbols with prior experience. Specific 
computer programs and interactive video programs are described. It is 
stressed that the computer programs being developed for literacy 
training for the hearing impaired should involve increasing the 
amount of linguistic interaction available to the child. (DB) 
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LITERACY AMD THE HEARING IMPAIRED: 
LIVING, MOVING, DYNAMIC TEXT 

Frank B. Wi throw, Ph.D* 

The hearing impaired person must rely upon the visual system 
for his or her major communication needs. Therefore, it is 
essential that we define the characteristics of visual perception 
and note weaknesses and strengths with respect to communication 
skills. If we compare auditory and visual sensory systems, the 
most obvious difference is that the auditory system is a global. 
That is, signals can reach the person from any direction, 24 hours 
a day. The auditory system is primarily an early warning system 
that is always "on". On the other hand, the visual system is 
directional rather than global and dependent upon sufficient light 
sources. To use the visual system the person must turn it on (open 
your eyes) and direct it to the source of the stimuli. 

The hearing person, of course, develops his language and 
communication skills through hearing and speech. The global nature 
of the hearing sensory system means that th^ infant is embedded in 
a sea of meaningful sounds that he can not turn off. Once language 
and speech patterns are established, they are transferred to the 
visual system through the development of literacy skills. 

The creation of written symbols was a giant leap forward for 
society. It meant that accurate, stable, and permanent records of 
the knowledge arid experience of civilization could be transferred 
over time and distance. Ideas developed in one part of the world 
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could be shared with others who never came in direct contact with 
the original thinkers* Moreover, ideas and events that happened 
in one generation could be shared with following generations. 

The primary language and communication skills of the average 
person are developed through speech and language and then 
transferred to the written form. As a beginning reader discovered, 
"Ohl Reading is just talk in a book*" This shows great insight on 
the part of the young learner. 

Since communication skills were built on oral and aural 
systems, certain safeguards have developed to assure clear communi- 
cations. There is redundancy built into the signal so that when 
the speech signal is buried in noise the receiver can fill in the 
blank spots. This is partially done by grammar and by phonetic 
clues . 

On the other hand, visual . communications, whether they are 
in the form of speech reading, finger spoiling, or sign language, 
have a number of ambiguities that act as noise in the signal. As 
noted earlier, the receiver must be looking at the sender and the 
light must be sufficient for vision to work. Obviously, 
communication skills do develop among the hearing impaired and 
there is no question that visual communication systems can be the 
primary system. Oral/aural, oral/visual, and manual/visual common 
communication systems are eventually associated with the more 
stable written format of language. 

The printed word developed differently in different parts 
of the world. One system was based upon pictographs and ideographs 



and the other on phonetic representations of the sounds. 
Pictographic and ideographic systems are still used. The average 
Japanese, for example, must learn several thousand different 
symbols to become literate. On the other hand, the nations that 
use phonetic alphabets must learn the letter tliat represents the 
sounds of the language. 

English spelling is a particular challenge to literacy since 
it is based upon a phonetic principle, but has many discrepancies. 
Consequently, the spelling of many English words must be learned 
by rote because in some instances words like "women" are not 
faithful phonetic representations. "GHOTI" for a new reader of 
English could logically be the spelling for FISH. They would 
associate the "gh" as the sound in enough, the "o" as the sound in 
women, and the "ti" as it is pronounced in nation. English 
spelling is simply not a faithful one-to-one phonetic language. 

On the other hand, finger spelling is a one-to-one manual/ 
written communication system that faithfully represents the printed 
word. Among other reasons for this problem in English was the 
invention of the printing press which fixed spelling in printers 
fonts and left us a legacy of non-phonetic spellings. English has 
44 phonemes which are represented by the 26 letters of our 
alphabet. But we do not consistently combine the letters to 
represent the sounds . Some sounds are represented by several 
different letter combinations. Simple examples c:re that both f and 
ph represent the "f" sound and that gh represents the "f" sound at 
the end of words. This causes problems in developing high levels 
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of literacy in hearing people. The United States has one of the 
highest illiteracy rates of any developed nation in the world. 

There have been many reformers who would have corrected our 
alphabet and given us a more precise system. Alexander Graham Bell 
developed an alphabet entitled visible speech. This was a thiee 
case system that provided information on where the lips, tongue, 
teeth, voice, breath, and nasal aspects of the phonemes were 
produced. This system was used extensively in transcribing 
languages that did not have a written form. George Bernard Shaw 
left his estate to the reform of English spelling to make it more 
phonetic, and Sir James Pitman has long sought a more scienti- 
fically based written system. Ironically, these reforms have been 
fought by the status quo of our society even though we pay a high 
price in illiteracy because of it. It just may be that the 
computer revolution will facilitate some of the needed reforras. 
Both synthesized speech and computer analyzed speech could be 
facilitated by a better alphabet and spelling system. 

We have been able to translate print into speech and are 
working on speech into print. There are available on the market 
today print to speech machines and promising speech to print 
machines . 

Barbara Tuckman was asked what were the most important 
technologies modern man had developed to increase literacy. Her 
reply, much to the dismay of computer and technology experts, was 
that in her opinion the greatest technologies to enhance universal 
literacy were (1) the electric light and (2) wearable eye glasses. 
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The electric light meant that reading could take place at any time 
during the 24 hour day and wearable glasses meant that people could 
read for their entire lives, regardless of the physical changes we 
all suffer with age. When you think about it this is a very 
profound answer. These technologies essentially expanded the 
available time on task for people to read. I have tried to set the 
stage for the use of print technologies in the education of hearing 
impaired youngsters by discussing the nature of literacy in 
general. In summary, the young hearing child develops through many 
associations within the environment a speech and language system 
that associates real experiences with speech symbols. Once that 
system is developed reading and writing are developed on top of the 
speech and listening system. 

The hearing impaired child, even if he or she is making 
maximum use of residual hearing, must develop a language based upon 
visual symbols. In most education programs there is a synergy 
among the spoken, signed, finger spelled word or language concept 
and the referent in the real world. As a matter of fact, during 
the development stages the printed symbol is also associated with 
the event. There are a number of studies that indicate that the 
printed word is the most stable of the visual symbols used by 
hearing impaired children. 

Technology provides us with a wide range of visual print 
symbols that can carry a richer meaning. For example, if we added 
color to closed captions we could assign a specific color for male 
voices and one for female voices. The captioning techniques of 
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today place the captions near the speakers. Voice over narration 
is indicated by placing the caption at the top of the screen. 
Music is indicated by "note quotes" around the sung portion of the 
program, etc. Obviously, there are a number of additional 
refinements that can be made. In a play, if male and female voices 
are coded by color, individual characters could be assigned 
different type fonts. J.R, in "Dallas" is always blue and with a 
geneva type font etc.. Captioning may be one of the most important 
services offered hearing impaired people. It provides them with 
a massive amount of language usage associated with story and action 
experiences on television. 

Language development remains a problem for hearing impaired 
infants and children. Traditionally we find that most hearing 
impaired children, regardless of the location of their school, 
remain three or more years behind their hearing peers in academic 
achievement. Much of this lag is dependent upon development of 
sophisticated language skills. Technology offers us the ability 
to provide new and different experiences that can associate 
language symbols with experience. For example, interactive video- 
discs or DVI CD-T^^' interactive programs can provide individual 
study stations that give the user a multiple linguistic experience. 
It is possible to develop a program that provides for language 
building experiences and allows the user to experience the program 
through auditory and speech symbols, finger spelled symbols, total 
communication systems, or American Sign Language at the flip of a 
switch. All can be associated with captions, 
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Text on screen is not limited to the traditional book text 
formats Words can pop on the screen, move, and indicate their 
meaning* For example, in "Sesame Street" and "The Electric 
Company" we had words that animated into their referents, FAT grew 
fat until it covered the entire screen, THIN became tall and thin. 
Other words such as angry , cry , happ , etc • , animated into 
characters that identified the state of feeling; verbs actually 
took on their meaning* FLY began to flap and fly around, WALK 
began to walk etc.. 

In the mid 1970 ' s, I was particularly interested in the 
ability of the computer to animate and change words into their 
referents. I took sentences, such as: The butterfly flew around 
the flower and as the words were said they changed into their 
meaningful referents, i.e., the word "butterfly" became a butterfly 
when it was said, "flew" (the word and the butterfly began to fly) , 
"around" began to circle and "the flower" turned into a flower. 
The final scene was a butterfly flying around a flower and the 
caption changed to: The butterfly is flying around the flower • 

In today's world of technology we might have this happen as 
the learner reads the sentence into the microphone or as the 
learner types it into the system. The beauty of today's technology 
is that it can be immediate and interactive. 

Margarat Withrow and Charles Csuri of Ohio State 
University's Computer Graphic Center created a series of computer- 
generated programs that allowed learners to experience a number of 
language concepts. These programs cue captions to auditory signals 
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and language concepts. 

Another computer-generated series of programs were 
experimented with at the National Technical Institute which in one 
instance gave the learner control over a "Star Trek" program • To 
save the universe from the Klingons the user had to work mathema- 
tical problems for the Captain. Another was a job interview. The 
last one used Withrow and Csuri created computer animated charac- 
ters. The learner could either describe what was happening or 
command the animals to do certain things within the limits^ of the 
computer's animated world. Most learners loved to command the 
animals. There was a certain degree of argumentative dialogue that 
went along with the program. For example, when the learner logged 
onto the program, a series of weather questions were asked and if 
the temperature was 20 below zero, snowing, and windy then the 
animals respond accordingly. If, for example, you ask the frog to 
swim, the computer asks you if there is an indoor pool, etc.. 
There is an Empire State building in the computer's animated 
environment. You can, of course, ask the frog to jxamp over the 
building. . .the frog will argue and finally attempt to jump, but of 
course hits the walls and ends the game. If you asked the frog to 
fly the computer might ask you how many wings the frog had or if 
you are willing to buy the frog an airline ticket. The learner 
entered the program through the keyboard. Today, or in the near 
future it would not be impossible to enter such programs through 
voice analysis. 

Interactive video programs could create a dictionary that 
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enabled the person to receive a moving example of the word, a 
spoken example of the word, a finger spelled example and a signed 
example of a word on demand of the learner. Such an interactive 
dictionary would be meaningful not only to hearing impaired 
students, but other learners who have not mastered literacy fully. 

'^uch of the work in applying technology to educate hearing 
impaired youngsters has been directed towards providing more 
linguistic experiences for them. In the 1960s, Robert Step at the 
University of ^Nebraska developed interactive 8mm films shot from 
the viewpoint of the young child and provided a number of 
manipulative activities for the young child and provided a number 
of manipulative activities for young deaf children to speech read. 
Withrow's Lip reading films were done in conjunction with the 
University of Nebraska and the Illinois State School for the Deaf. 
These were single concept films developed to provide experiences 
and increase vocabulary. PROJECT LIFE was developed to again 
provide language experiences through technology. 

John Gough, Gilbert Delgado, and Malcolm J. Nowrood, through 
their leadership in captioning techniques, explored many avenues 
for creating more new linguistic experiences for the hearing 
impaired population. George Propp and Robert Step were pioneers 
at the University programs and were on the cutting edge of design 
for interactive videodiscs. 

The key to captioned television for the hearing impaired and 
the development of computer programs for literacy for the hearing 
impaired is that they increase significantly the amount of 
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linguistic interaction available to a young hearing impaired child. 

If we look at the history of education of the hearing 
impaired children, we can see that many of our forbearers explored 
a wide range of technological and scientific applications to 
education of the hearing impaired. Obviously, Alexander Graham 
Bell contributed much in this area through the telephone; however, 
his understanding of speech, literacy, and print may have been a 
greater contribution. Many of the pioneers of education of the 
deaf, such as Edith Fitzgerald and her peers, developed 
technologies of form and structure that assisted in the development 
of language for the deaf. 

The Regional Media Centers for the Deaf and their drive to 
teach teachers of the hearing impaired to use among other things, 
the overhead projector as a blackboard increased the teacher-pupil 
interface time. A teacher of the deaf who turns to the blackboard 
to write loses contact with the pupil. A teacher using an overhead 
projector maintains contact. 

This year will mark forty years that I have been interested 
in the use of technology to reinforce education of hearing impaired 
youngsters. In that period of time, we have seen great advances 
in technologies. Today, we can develop a wide range of universally 
available learning tools that will increase the quality of 
education for all hearing impaired people. Today, we have a mature 
captioned television program with more than 2 00 hours of available 
programs that we can view each week. We have the opportunity to 
expand that service by looking at more efficient formats of 
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captioning. There are a nuiaber of excellent captioned educational 
materials available for the classroom teacher to use. 

I would like to see greater development of computer and 
interactive laser technologies used in education of the hearing 
impaired. 

I would like to see a more rapid conversion of TDD systems 
to ASCII codes so that the hearing impaired will have more 
universal access to electronic systems. I would like the 
development of more sophisticated uses of language in interactive 
programs. There is a continuing need for upgrading teachers' 
skills in using new technologies. I would like to see programs 
take advantage of distance learning technologies to reach the 
professional community. 

There is much to do and it should be challenging. If we 
solve some of the literacy problems among the hearing impaired, we 
may also be working to solve literacy problems in the total 
community. We can ill afford to remain a nation that does not work 
to make the best of all of our human talent. We cannot allow 
significant numbers of our citizens to remain illiterate. 

The door to achievement for hearing impaired people is the 
development of good literacy skills. That is the ability to read 
with meaning and to write with a critical eye to the knowledge they 
have. 
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